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> A note-taking system based on a notepad 
computer with an integrated audic/video-recor- 
der is described. A document is created or 
retrieved. As the user types on the keyboard or 
writes with the stylus or simBar input instru- 
ment each character or stroke that is input by 
(he user is invisibly time-etamped by the com- 
puter. The audtorVideo stream is also continu- 
ously time-stamped during recording. To play a 
section of recording back, the user selects part 
of the note (perhaps by drding 1 with a stylus) 
and invokes a "playback selection* command. 
The computer then examines the time-stamp 
and "winds" the record to the corresponding 
place in the audio/video recording, where it 
starts playing - so that the user hears and/or 
sees what was being recorded at the instant the 
selected text or strokes were input With a 
graphical user interface, the user may input key 
'topic' words and aubeequenfly place check 
marks by the a p propriate word as the convers- 
ation topic veers into that neighborhood. 
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This invention relate to a data access system, 
and in perticUar to one in which the taking of note 
usualy accompanies the recording of data. The note 
themeelves are used to gain selective access to the 
data. 

The problem 

At seminars, interviews or meetings, it has long 
been recognaed that the simple act of taking notes 
helps the writer memorize key facts. On the other 
hand, to make a note, the listener has to divert atten- 
tion from the speaker, figure out how to encode the 
in for mati on he has heard or the idea he has had, and 
then focus on writing it down. ttdoesnl seem surpris- 
ing that the listener often loses the speaker's thread! 
Nowadays many people resort to making audio or 
video recor di n g s of important events and then spend- 
ing time transcribing key ideas afterwards. Locating 
the interesting parts of an audio or video recording is 
often very tim e - c onsuming and tedious, which 
reduces the fikelihood that the listener wil bother to 
transcribe the tape, or even bother to make the 
recording in tie first place. 

In many situations the need to record is not anti- 
cipated Only after the event, sometimes long after- 
wards, does the need become evident To overcome 
this problem it may eventually become technologi- 
cally possible, from a storage point of view, to record 
and store every second of a person's working day. 
The problem of locating key pieces of information from 
this huge and expanding base of recordings becomes 
overwhelming. 

This invention buBds upon these ideas by provid- 
ing a semi-automatic fine-grained, audktfvkJeo index- 
ing tool. Although the invention focuses on gaining 
access to audio and video records, it is not limited to 
this application. It is envisaged that the invention 
could also be applied to accessing ties in a computer 
system, phone call records, or any other time-stam- 
ped data set 

The present invention w9 now be described by 
way of example with reference to the accompanying 
drawings, in which; 

Figure 1 tea diagrammatic view of one form of the 

invention using a laptop computer; 

Figure 2 is a diagrammatic view of another form 

of the invention using a notepad computer with 

stylus input and 

Figure 3 is a diagrammatic view of one form of the 

archtecfcire of the invention. 

The example of the invention shown in Figure 1 
combines a laptop computer 2 with a video or audio 
recording device 4. 

In the example of the invention shown in Figure 
2, the portable computer is replaced by a note-pad 
computer 6 with stylus or other graphical input device 
8, connected to video or audio recording device 4. The 
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stylus inputdevkxnwytaketeplaceoCthekeybcMJtr 1 
of the portable computer as the user's means of input 
Alternatively, a keyboard 1 0 may be connected to the 
notepad computer for input 

5 Either form of computer is interfaced to the recor- 
der in such a manner that the computer can monitor 
the recorder's progress during recording. To enable 
this, the recorder may emit time code or make other 
unique index i nfo r mat ion continuously avaiable to the 

io computer. Additionally the comp u ter can operate the 
recorder directly, causing it to record, or play from a 
specific index point as required. In any sftuation it is 
possible for the computer to find out the time-code or 
index information. 

15 In the general system shown in Figure 3, the com- 
puter presents a document editor style user interface 
to the user. This interface may resemble that of a 
word-processor, an ilustrator, or an application speci- 
fic interface, depending on the user's needs and com- 

20 puter hardware. The user interface provides 
commands for creating a new document and starting 
and stopping recording. 

A new document is created, or an old one recal- 
led, and recording commences. The editor 22 allows 

2$ the user to draw, type or sketch on the blank docu- 
ment using keyboard 10, stylus 8, or other graphical 
input device. As each mark (or indicium) 26, is added 
to the document (an 'indicium' is any indivisible dis- 
playable symbol created by the interface, e.g. pen- 

30 stroke, character, graphical symbol, etc), the editor 
time-stamps and stores it in an indicium-to-timestamp 
index 14. The video-frame time-stamper 20 also con- 
tinuously notes the index information 32 arriving from 
the recorder 4 at that instant and time-stamps it and 

35 stores it in a timestamp-to-timecode index 16. The ti- 
me-stamps are not visible to the user - they are stored 
with the computer's internal representation of the indi- 
cia When the session is over, the user stores away 
the document the contents of the two indexes, and 

40 the recording for subsequent use. Any method that 
allows time-stamps to be used as an Max into the 
recording may be used instead of the timestamp-to-ti- 
mecode index, e.g. interpolation of the timecode itself. 
When the user wants to recal sections of the 

4$ recording, the appropriate related document is ret- 
rieved and recalled to the display 30 by the browser 
24. Using the stylus 8, or the keyboard 10, the user 
selects one or more indicia on the document perhaps 
by circling them, positioning a cursor, or typing an 

so identifying name, and instructs the browser 24 to play 
the associated video section. The browser 24 iden- 
tifies the selected indicium or indicia, looks up the ti- 
me-stamp(s) in the indicium-to-timestamp index 14; 
looks up the timestamp(s) in the timestamp-to-timeco- 

55 de index 16, and plays the section of the recording in 
the area indicated by the resultant timecode. Thus the 
user sees what was recorded at the time the marks 
were made on the video monitor 28. 
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IndiCHim^o-timestarnp and timestamp-to-timeco- 
de indices may be combined into a single tnditium-tD- 
b me code index. 

K is expected that users wil develop shorthand 
notations for marking the document to indicate an 5 
interesting idea, change of topic or speaker, and so 
forth. It also is expected that users wil type or hand- 
write key words aa the topic arises, and then add 
additional marks nearby each time the subject veers 
in the direction of that topic Thus, to hear all section 10 
of a seminar associated wih the same topic it is 
necessary only to ctrde all the marks in the location 
of the topic keyword. 

Accordingly it will be seen that the present inven- 
tion provides a simple, inexpensive, lightweight, low- 15 
power device for quick access to data records. 



Claims 

20 

1. A system for providing random access to a time- 
stamp identified part of a time-stamped data set 
said system comprising: 

means for capturing, filing, retrieving, dis- 
playing, and automatically time-stamping seer- 25 
generated indicia for selection by a user, said 
time-stamped data set and said time-stamped 
indicia having a common time base; 

means for determining the time-stamp of 
each user selected indicia; and 30 

means for indexing into said data set to 
identify the part thereof that is time-stamp corre- 
lated with said selected indicia. 

2. A system as claimed in daim 1 , induding means 35 
for presenting to the user the identified portion of 

the data set 

3. A system as daimed in daim 2, in which the time- 
stamped data set is audio and/or video data. 40 

4. A system as claimed in any of claims 1 -3, which 
is implemented by means of a portable computer. 

5. A system as daimed in any of daims 1 -3, which 4$ 
is implemented by means of a portable video 
recorder. 

6. A system as claimed in claim 5, in which the port- 
able video recorder is a camcorder. so 

7. A system as daimed in daim 4 in which the port* 
able computer is a notepad computer with stylus 
input 
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